Multimodal Communication in the Virtual Farm of the Staging Project

1 Abstract This paper describes the multimodal communication components of a 3D virtual world application that is being developed within the Danish research project Staging. The application is a farm scenario in which the user can interact with an autonomous farmer agent via an avatar to participate in simple conversations relating to life and work on the farm. Communication is multimodal in that the agent can respond to verbal input combined with a limited number of hand gestures, and it is embedded in a mixed-initiative dialogue model based on dialogue obligations and goals. In addition to interacting with the user’s avatar, the agent reacts to other stimuli in the world, and its course of action – for instance whether or not to cooperate with the user – will ultimately depend on the relevance of specific behaviours at any point in time.